Test-Only Learning via Virtual Patients to Improve Surgical Illness Scripts of Preclinical Medical Students as a Preparation for Clinical Clerkships: An Experimental Study.

OBJECTIVE: The aim was to determine whether preclinical medical students can develop their illness scripts to a level comparable to that of clerkship students through test-only learning using repeated formative online testlets. METHODS: In this experimental study, participants were 52 preclinical and 53 clinical medical students. The intervention group consisted of preclinical medical students, and the control group consisted of clinical medical students. The intervention group responded to online testlets containing feedback, an innovative formative assessment method called ContExtended Questions, on general surgery for 8 days by spending no more than 30 minutes each day. The control group completed the general surgery clerkship. The performances were assessed using 20 Key-Feature Question items. The intervention group was assessed twice: immediately after the intervention (the immediate test), and again 1 month later (the delayed test). The control group was assessed once, immediately after the clerkship. All performance tests were identical. RESULTS: The preclinical students had a significantly higher mean score on the immediate test (83.1 ± 9.6) compared to the clinical students (75.4 ± 8.9), P < .001. The effect size (Cohen's d) was .83. However, the mean score in the delayed test (76.9 ± 13.6) was not significantly different from clinical students' mean score (75.4 ± 8.9), P > .05. CONCLUSIONS: Test-only learning as a spaced repetition of online formative testlets is effective in preparing preclinical medical students to the clinical clerkship. Through using this approach in preclinical period, they can prepare themselves for the clinical environment to optimize the benefits derived from clerkships.

Why teachers do (or do not) implement recommended teaching practices? An application of the theory of planned behavior.

Introduction: In Luxembourg, competency-based practices (CBP), differentiated instruction (DI), and formative assessment (FA) have been imposed by the 2009 school law. Referring to the Theory of Planned Behavior (TPB), this study examined factors influencing the implementation of these practices in classrooms. Methods: Teachers participated in an online survey assessing their attitudes, subjective norm, perception of behavioral control, intention, and pedagogical practices regarding CBP, DI, or FA. Measurement models were used in structural equation models testing the TPB. Results: If the main relationships postulated by the theory were confirmed, some inconstancies were observed depending on the targeted practices. Structural equation TPB models controlling for gender, experience, teaching level, and socio-economic level of the school population explained between 20 and 45% of the variance in teachers' practices, and between 65 and 75% of the variance in teachers' intention to use these practices. Discussion: The relevance of the TPB for studying teaching practices and implications for professional training are discussed.

Correlation Between Student Performances on Case-Based Constructed-Response Formative Assessment and Summative Assessment.

OBJECTIVES: This study aimed to evaluate the impact of formative assessment with case-based constructed-response question (CRQ) formats on student performance on the final summative assessment in the second-year periodontics course. METHODS: Classroom quizzes with case-based CRQs were implemented as the formative assessment during the course. Each student received feedback on their responses from the course director. After all students (N = 128) took the second-year final examination, the Friedman test was conducted to compare student performances in each assessment over time. The multiple linear regression (MLR) model was used to evaluate the association between the second-year final examination score and plausible predictors-student gender, the second-year formative and midterm examination scores, and time spent on the final examination. RESULTS: The mean % scores in the formative assessment (51) and midterm (84) examination were significantly lower than that of the final (87) examination (P < .01). The number of students who failed the final (6) examination was significantly lower than the midterm (16) examination (P = .03). The midterm (P < .0001) and the formative assessment (P = .0009) scores significantly affected the second-year final examination score while student gender (P = .59) and time spending (P = .83) showed no correlations. CONCLUSION: Within the limitations of the study, student performance on case-based CRQs was correlated with student performance on the summative assessment.

Does 'summative' count? The influence of the awarding of study credits on feedback use and test-taking motivation in medical progress testing.

Despite the increasing implementation of formative assessment in medical education, its' effect on learning behaviour remains questionable. This effect may depend on how students value formative, and summative assessments differently. Informed by Expectancy Value Theory, we compared test preparation, feedback use, and test-taking motivation of medical students who either took a purely formative progress test (formative PT-group) or a progress test that yielded study credits (summative PT-group). In a mixed-methods study design, we triangulated quantitative questionnaire data (n = 264), logging data of an online PT feedback system (n = 618), and qualitative interview data (n = 21) to compare feedback use, and test-taking motivation between the formative PT-group (n = 316), and the summative PT-group (n = 302). Self-reported, and actual feedback consultation was higher in the summative PT-group. Test preparation, and active feedback use were relatively low and similar in both groups. Both quantitative, and qualitative results showed that the motivation to prepare and consult feedback relates to how students value the assessment. In the interview data, a link could be made with goal orientation theory, as performance-oriented students perceived the formative PT as not important due to the lack of study credits. This led to low test-taking effort, and feedback consultation after the formative PT. In contrast, learning-oriented students valued the formative PT, and used it for self-study or self-assessment to gain feedback. Our results indicate that most students are less motivated to put effort in the test, and use feedback when there are no direct consequences. A supportive assessment environment that emphasizes recognition of the value of formative testing is required to motivate students to use feedback for learning.

Improving assessment of procedural skills in health sciences education: a validation study of a rubrics system in neurophysiotherapy.

BACKGROUND: The development of procedural skills is essential in health sciences education. Rubrics can be useful for learning and assessing these skills. To this end, a set of rubrics were developed in case of neurophysiotherapy maneuvers for undergraduates. Although students found the rubrics to be valid and useful in previous courses, the analysis of the practical exam results showed the need to change them in order to improve their validity and reliability, especially when used for summative purposes. After reviewing the rubrics, this paper analyzes their validity and reliability for promoting the learning of neurophysiotherapy maneuvers and assessing the acquisition of the procedural skills they involve. METHODS: In this cross-sectional and psychometric study, six experts and 142 undergraduate students of a neurophysiotherapy subject from a Spanish university participated. The rubrics' validity (content and structural) and reliability (inter-rater and internal consistency) were analyzed. The students' scores in the subject practical exam derived from the application of the rubrics, as well as the rubrics' criteria difficulty and discrimination indices were also determined. RESULTS: The rubrics´ content validity was found to be adequate (Content Validity Index > 0.90). These showed a unidimensional structure, and an acceptable internal consistency (α = 0.71) and inter-rater reliability (Fleiss' ƙ=0.44, ICC = 0.94). The scores of the subject practical exam practically covered the entire range of possible theoretical scores, showing all the criterion medium-low to medium difficulty indices - except for the one related to the physical therapist position-. All the criterion exhibited adequate discrimination indices (rpbis > 0.39), as did the rubric as a whole (Ferguson's δ = 0.86). Students highlighted the rubrics´ usefulness for learning the maneuvers, as well as their validity and reliability for formative and summative assessment. CONCLUSIONS: The changed rubrics constitute a valid and reliable instrument for evaluating the execution quality of neurophysiotherapy maneuvers from a summative evaluation viewpoint. This study facilitates the development of rubrics aimed at promoting different practical skills in health-science education.

Evaluation of student readiness for clinical dental education at the end of preclinical education.

OBJECTIVES: This study examined results of a summative objective structured clinical examination (OSCE) at the end of preclinical periodontal education to identify deficient areas for dental students in patient care and to explore factors affecting the probability of passing the OSCE. METHODS: The summative OSCE was administered to two consecutive cohorts, Classes A (class of 2024; n1 = 134) and B (class of 2025; n2 = 129). The questions for each station in the OSCEs were available to both classes 1 week before the OSCEs. Descriptive statistics were used to identify deficient areas. The multiple logistic regression model was built to predict the probability of passing the OSCE based on the cohort, gender, and the practical and written examination scores. RESULTS: Fifty-one (38%) students in Class A and 66 (51%) students in Class B completed the OSCE by passing all stations. Students undergoing remediation showed deficiencies in demonstrating how to detect tooth mobility, performing periodontal probing, drawing the healthy positive bony architecture and the mucogingival junction, and using a universal and a Gracey 13/14 curette. The probability of passing the OSCE was significantly correlated with Class B (p = 0.035) and the practical examination score (p = 0.03) while not associated with gender (p = 0.53) and the written examination score (p = 0.11). CONCLUSION: Students showed deficiencies in assessment skills at the end of preclinical education. The study findings suggest that the implementation of the OSCE at the conclusion of preclinical education would be beneficial since the written examination score might not accurately reflect student readiness for clinical patient care.

The Impact of Script Concordance Testing on Clinical Decision-Making in Paramedic Education.

This study investigates the effectiveness of the Script Concordance Test (SCT) in enhancing clinical reasoning skills within paramedic education. Focusing on the Medical University of Lublin, we evaluated the SCT's application across two cohorts of paramedic students, aiming to understand its potential to improve decision-making skills in emergency scenarios. Our approach, informed by Van der Vleuten's assessment framework, revealed that while the SCT's correlation with traditional methods like multiple-choice questions (MCQs) was limited, its formative nature significantly contributed to improved performance in summative assessments. These findings suggest that the SCT can be an effective tool in paramedic training, particularly in strengthening cognitive abilities critical for emergency responses. The study underscores the importance of incorporating innovative assessment tools like SCTs in paramedic curricula, not only to enhance clinical reasoning but also to prepare students for effective emergency responses. Our research contributes to the ongoing efforts in refining paramedic education and highlights the need for versatile assessment strategies in preparing future healthcare professionals for diverse clinical challenges.

Feasibility, usability, and acceptability of MobileCoach-Teen: A smartphone app-based preventative intervention for risky adolescent drinking behavior.

BACKGROUND: Older adolescence (ages 15-18) is a critical period for experimentation with substance use, especially alcohol. Adolescent drinking poses hazards to physical and mental health, amplifies risk associated with other activities typically initiated during this life stage (e.g., driving, sexual activity), and is associated with adverse outcomes in adolescence and adulthood. Existing preventative interventions are expensive and have questionable long-term efficacy. Digital interventions may represent an accessible and personalized approach to providing preventative intervention content to youth. METHODS: This study recruited 29 adolescents aged 16-18 (M = 17.24, SD = 0.74) for a pilot feasibility trial of the MobileCoach-Teen (MC-Teen) smartphone app-based intervention. The study team randomized participants to receive either the alcohol intervention (MC-Teen) or attention control pseudo-intervention (MC-Fit). MC-Teen participants received 12 weeks of content adapted from a prior Swiss-based trial of a preventative alcohol intervention. Participants provided qualitative and quantitative feedback at baseline, via six biweekly surveys during and post-intervention. RESULTS: Both groups rated the application as easy to download (M = 4.31, SD = 0.93; 5-point Likert). All participants completed the baseline survey in less than the estimated time of 10 min (M = 7:42, SD = 2:15) and rated the survey as easy to complete (M = 4.69, SD = 0.60; 5-point Likert). MC-Teen participants favorably assessed application user experience, message user experience, and digital working alliance with application. Qualitative themes included a desire for increased rate/amount and diversity of content, greater representation via coach options, user interface/user experience improvements, and additional features. CONCLUSION: The MC-Teen intervention is feasible and acceptable based on a pilot feasibility trial with a sample of U.S. adolescents.

First steps of learning analytics in a blended learning general practice curriculum at Saarland University - a quantitative approach.

Objectives: Medical education has been revolutionized by the growing importance of digital learning. Little is known about students' online study behaviour and its relationship with exam performance. This quantitative study analyses and describes students' digital learning behaviours in a blended learning curriculum for General practice at Saarland University, Germany. It also examines the relationship between digital learning behaviour and exam performance. Methods: Cohort and individualized AMBOSS® user data from 195 students at Saarland University was analysed quantitatively. Performance in course-specific multiple-choice question sessions and user data of the integrated online learning activities were correlated with each other and with General practice exam grades. Anonymized data from 10,534 students from 35 other German universities served as the reference cohort. Differences in digital learning behaviour between the groups were calculated using Mann-Whitney-U-Test for non-normally distributed data. Results: Students in the blended learning course used integrated content more frequently than the reference cohort (U=48777, p<0.001). The number of digital learning cards read correlated moderately with digital formative assessment performance (ρ=0.331, p=0.005 and ρ=0.217, p=0.034). Formative assessment scores and exam results correlated strongly in the summer semester cohort (ρ=0.505, p<0.001), and moderately in the winter semester cohort (ρ=0.381, p<0.001). Conclusion: There is a difference in the usage of online learning activities when they are purposefully integrated into a curriculum. Digital learning activities including formative assessment may serve as valuable, constructively aligned exam preparation. This is relevant for medical educators when planning future blended learning curricula and portfolio systems, as it may save financial and human resources.

Formative behavioral assessment across eight constructs: Dependability of direct behavior ratings and formative behavior rating measures.

Due to their promise as a feasible tool for evaluating the effects of school-based interventions, Direct Behavior Ratings (DBR) have received much research attention over the past 2 decades. Although DBR methodology has demonstrated much promise, favorable psychometric characteristics only have been demonstrated for tools measuring a small number of constructs. Likewise, although a variety of methods of DBR have been proposed, most extant studies have focused on the use of single-item methods. The present study examined the dependability of four methods of formative behavioral assessment (i.e., single-item and multi-item ratings administered either daily [DBR] or weekly [formative behavior rating measures or FBRM]) across eight psychological constructs (i.e., interpersonal skills, academic engagement, organizational skills, disruptive behavior, oppositional behavior, interpersonal conflict, anxious depressed, and social withdrawal). School-based professionals (N = 91; i.e., teachers, paraprofessionals, and intervention specialists) each rated one student across all eight constructs after being assigned to one of the four assessment conditions. Dependability estimates varied substantially across methods and constructs (range = 0.75-0.96), although findings of the present study support the use of the broad set of formative assessment tools evaluated.

Automated feedback and writing: a multi-level meta-analysis of effects on students' performance.

Introduction: Adaptive learning opportunities and individualized, timely feedback are considered to be effective support measures for students' writing in educational contexts. However, the extensive time and expertise required to analyze numerous drafts of student writing pose a barrier to teaching. Automated writing evaluation (AWE) tools can be used for individual feedback based on advances in Artificial Intelligence (AI) technology. A number of primary (quasi-)experimental studies have investigated the effect of AWE feedback on students' writing performance. Methods: This paper provides a meta-analysis of the effectiveness of AWE feedback tools. The literature search yielded 4,462 entries, of which 20 studies (k = 84; N = 2, 828) met the pre-specified inclusion criteria. A moderator analysis investigated the impact of the characteristics of the learner, the intervention, and the outcome measures. Results: Overall, results based on a three-level model with random effects show a medium effect (g = 0.55) of automated feedback on students' writing performance. However, the significant heterogeneity in the data indicates that the use of automated feedback tools cannot be understood as a single consistent form of intervention. Even though for some of the moderators we found substantial differences in effect sizes, none of the subgroup comparisons were statistically significant. Discussion: We discuss these findings in light of automated feedback use in educational practice and give recommendations for future research.

Effective Feedback Strategy for Formative Assessment in an Integrated Medical Neuroscience Course.

Purpose: Despite the different benefits of formative assessments in an integrated medical curriculum, the effective strategies to provide feedback to medical students to benefit from the different merits of formative assessment are not fully understood. This study aims to determine the effect of different strategies of formative feedback on students' outcomes in a medical neuroscience course. Method: We compared medical students' performance in summative examinations in the academic year that formative feedback was provided using in-person discussion and compared such performances with the academic year when the feedback was provided by written rationales or a combination of written rationales and in-person discussion. We also surveyed medical students' preferences for whether written or in-person formative feedback is a better strategy to provide feedback at the end of each course. Results: ANOVA found a significant difference in summative performance scores for those scoring ≥ 70% when formative feedback was provided by providing a rationale, in-person, and a combination of both ([F (2,80) = 247.60, P < 0.001]. Post hoc analysis revealed a significant and highest performance when feedback was provided using the written rationale approach (***P < 0.05), followed by in-person (**P < 0.05). In contrast, the least performance was recorded when formative feedback was provided using a combination of providing a written rationale for the answers to the questions and in-person discussion of the questions (*P < 0.05). Students' preferred approach for receiving formative feedback for their formative assessment was highest for written rationale (***P < 0.05), followed by in-person or a combination of in-person and written rationale (**P < 0.05). Conclusion: Our results found that medical students preferred a written formative feedback approach, which was associated with better student performance on the summative examination. This study reveals the importance of developing effective strategies to provide formative feedback to medical students for medical students to fully benefit from the merits of formative assessment in an integrated medical school curriculum.

A qualitative study examining the critical differences in the experience of and response to formative feedback by undergraduate medical students in Japan and the UK.

BACKGROUND: Formative feedback plays a critical role in guiding learners to gain competence, serving as an opportunity for reflection and feedback on their learning progress and needs. Medical education in Japan has historically been dominated by a summative paradigm within assessment, as opposed to countries such as the UK where there are greater opportunities for formative feedback. How this difference affects students' interaction with feedback has not been studied. We aim to explore the difference in students' perception of feedback in Japan and the UK. METHODS: The study is designed and analysed with a constructivist grounded theory lens. Medical students in Japan and the UK were interviewed on the topic of formative assessment and feedback they received during clinical placements. We undertook purposeful sampling and concurrent data collection. Data analysis through open and axial coding with iterative discussion among research group members was conducted to develop a theoretical framework. RESULTS: Japanese students perceived feedback as a model answer provided by tutors which they should not critically question, which contrasted with the views of UK students. Japanese students viewed formative assessment as an opportunity to gauge whether they are achieving the pass mark, while UK students used the experience for reflective learning. CONCLUSIONS: The Japanese student experience of formative assessment and feedback supports the view that medical education and examination systems in Japan are focused on summative assessment, which operates alongside culturally derived social pressures including the expectation to correct mistakes. These findings provide new insights in supporting students to learn from formative feedback in both Japanese and UK contexts.

Design of online formative assessment of nursing humanities curriculum during the COVID-19 pandemic: A teaching practice research.

BACKGROUND: Under the digital transformation trend nursing education, online formative assessment (OFA) provides a new opportunity. However, the OFA of nursing humanities course lacks design and practice, and faces the challenge of enhancing effective communication between teachers and students, student participation and autonomous learning. OBJECTIVES: To enhance the reliability of OFA in nursing humanities courses and provide practical experience for online teaching in the nursing profession. DESIGN: A quantitative research approach was used. SETTING: This study was conducted in a comprehensive university in China. PARTICIPANTS: We conducted teaching practice on 185 nursing undergraduates, with 89 students in the experimental group, and 96 students in the control group. METHODS: In the 2020-2021 multicultural nursing course, student learning outcomes and questionnaires were analyzed through the online learning tool Superstar Learning, student feedback and satisfaction questionnaires, and descriptive analysis and independent sample t-tests were conducted using SPSS 25.0 software. RESULTS: The OFA of students using Superstar Learning differed in learning performance and time to receive feedback from teachers between the experimental and control groups, and both groups had higher satisfaction levels. The experimental group's instructional design contained a synchronous classroom discussion module with better participation. CONCLUSIONS: During the COVID-19 pandemic, the use of online learning tools can support the implementation of OFA, build an environment where teachers and students participate together, have a positive impact on the continuous updating of teachers' teaching programs and students' learning outcomes. Simultaneous classroom discussions are expected to be an effective way to improve the reliability of OFA. Our instructional design, provides best practice suggestions for future online teaching and learning.

Formative assessment and student understanding of structure-function.

The structure and function relationship is a core concept identified by physiology faculty. Prior research has shown this may be a difficult concept for students to understand. Formative written assessments, such as short answer essay questions, allow students to demonstrate their thinking by encouraging students to use their diverse ideas to construct their responses. Varying the context of a question, such as the inclusion of a scenario, may be used to provide insight into the different stages of students' emerging biological expertise. Short answer questions based on the core concept structure↔function were administered to students in a junior level General Physiology course and a sophomore level Human Anatomy and Physiology course at a large southeastern public university. Questions were based on the integumentary, muscular, digestive, and cardiovascular systems. Student responses were scored with a conceptual rubric developed for each question prompt as well as each organ system represented in the question prompts. Students were interviewed to determine if their responses to the short answer questions accurately reflected their thinking. Less than half of the student responses in this study demonstrated a conceptual understanding of the structure-function relationship. Students demonstrated different conceptual understanding of structure↔function concepts depending on the question prompt with a scenario versus the question prompt without a scenario. The question prompts with scenarios versus non-scenarios provided a different context, which may have influenced student explanations. These results suggest that instructors should provide students with questions in varying contexts to allow students to demonstrate their heterogeneous ideas about a concept.NEW & NOTEWORTHY Formative assessment provides feedback to both students and instructors about the process of learning. The core concept structure-function provides a foundation upon which many topics in anatomy and physiology can be built across all levels of organization. This study contributes to existing research about student conceptual understanding of the core concepts. Implications for practitioners include the design of formative assessments and a suite of questions to be used to gauge student understanding of structure-function.

FoCo: A Shiny App for Formative Assessment using Cognitive Diagnosis Modeling / [FoCo: una aplicación Shiny para la evaluación formativa usando modelos de diagnóstico cognitivo]

Combining formative and summative evaluations could improve assessment. Cognitive diagnosis modeling (CDM) has been proposed as a tool for diagnosing students’ strengths and weaknesses in formative assessment. However, there is no user-friendly software to implement it. For this reason, a Shiny app, FoCo, has been developed (https://foco.shinyapps.io/FoCo/), to conduct CDM and classical test theory analyses. The responses from 86 undergraduate students to a research methods course examination were analyzed. Students’ strengths and needs were diagnosed concerning their dominance of the syllabus contents and the first three competencies in Bloom’s taxonomy. The validity of the results was analyzed. The exam showed acceptable about evaluating students’ knowledge, as students with similar scores showed different strengths and weaknesses. Additionally, these attributes were found to predict different relevant criteria. It is expected that FoCo’s easiness to use promotes the employment of CDM in real educational settings.(AU)

La combinación de evaluaciones formativas y sumativas podría mejorar la evaluación. El modelado de diagnóstico cognitivo (MDC) se ha propuesto para diagnosticar fortalezas y debilidades de estudiantes en la evaluación formativa. Sin embargo, ningún software permite implementarlo fácilmente. Así, se ha desarrollado FoCo (https://foco.shinyapps.io/FoCo/), permitiendo realizar análisis MDC y teoría clásica de tests. Se analizaron respuestas de 86 estudiantes de grado a un examen de métodos de investigación, diagnosticándose sus fortalezas y necesidades en cuanto a su dominio de los contenidos de la asignatura y las tres primeras competencias de la taxonomía de Bloom y se analizó la validez de los resultados. El análisis ha sido informativo, ya que para estudiantes con puntuaciones similares ha sido posible detectar diferentes fortalezas y debilidades. Además, se encontró que estos atributos predicen criterios relevantes. Se espera que FoCo facilite el uso de MDC en contextos educativos.(AU)

Impact of Audience Response System in Enhancing Teaching of Anatomy and Physiology for Health Sciences Students at King Saud bin Abdulaziz University for Health Sciences.

Purpose: Instructional methods that offer students' opportunity for engagement are essential to develop an active learning environment. This paper aims (1) to assess whether the use of an Audience Response System (ARS) during anatomy and physiology lectures improves students' engagement, knowledge retention and academic achievement and (2) to evaluate the feasibility of introducing ARS as a formative tool from both instructors' and students' perspectives. Materials and Methods: This quasi-experimental study was conducted across ten lectures for the second-year Pre-Applied Medical Science (PAMS) and Pre-Medical (PMED) students at the College of Sciences and Health Professions, King Saud bin Abdulaziz University for Health Sciences (KSAU-HS), Jeddah, Saudi Arabia. The ARS was integrated into five lectures, whereas the remaining were conducted without ARS. A comparison of the immediate post-lecture and prior laboratory session quiz scores between the lectures with and without ARS was performed using an independent sample t-test. An online survey was completed by the students as well as informal feedback was taken from the instructors to assess ARS usefulness. Results: A total of 65 PMAS and 126 PMED students participated in the study. Students achieved significantly better scores for ARS lectures than non-ARS lectures: PAMS (p 0.038) and PMED (p 0.018). Students and instructors agreed that ARS was easy to use, allowing students to engage and actively participate in the teaching process by responding to questions and receiving instant and anonymous feedback about their learning progress. Conclusion: Employing suitable interactive teaching methods facilitates students' learning and improves their retention of knowledge. Students and instructors positively view the ARS strategy as a means to promote learning in a traditional lecture setting. Increased practice on how to integrate its use into classrooms could lead to further utilization.